{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9fd54a32",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/llm/openai_responses.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e3a8796-edc8-43f2-94ad-fe4fb20d70ed",
   "metadata": {},
   "source": [
    "# OpenAI Responses API\n",
    "\n",
    "This notebook shows how to use the OpenAI Responses LLM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081d07d2",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f6f8702",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index llama-index-llms-openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b007403c-6b7a-420c-92f1-4171d05ed9bb",
   "metadata": {},
   "source": [
    "## Basic Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0a6c491",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b79b0d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    # api_key=\"some key\",  # uses OPENAI_API_KEY env var by default\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ead155e-b8bd-46f9-ab9b-28fc009361dd",
   "metadata": {},
   "source": [
    "#### Call `complete` with a prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60be18ae-c957-4ac2-a58a-0652e18ee6d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "resp = llm.complete(\"Paul Graham is \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac2cbebb-a444-4a46-9d85-b265a3483d68",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Graham is a prominent computer scientist, entrepreneur, and venture capitalist, best known for co-founding the startup accelerator Y Combinator. He is also recognized for his essays on technology, startups, and programming, which have influenced many in the tech community. Graham has a background in programming languages and artificial intelligence, having earned a Ph.D. from Harvard University. His work has significantly shaped the startup ecosystem, particularly in Silicon Valley. Would you like to know more about a specific aspect of his work or ideas?\n"
     ]
    }
   ],
   "source": [
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14831268-f90f-499d-9d86-925dbc88292b",
   "metadata": {},
   "source": [
    "#### Call `chat` with a list of messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbe29574-4af1-48d5-9739-f60652b6ce6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"system\", content=\"You are a pirate with a colorful personality\"\n",
    "    ),\n",
    "    ChatMessage(role=\"user\", content=\"What is your name\"),\n",
    "]\n",
    "resp = llm.chat(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cbd550a-0264-4a11-9b2c-a08d8723a5ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: Ahoy, matey! Ye can call me Captain Jollybeard, the most colorful pirate to sail the seven seas! What brings ye to me ship today? Arrr!\n"
     ]
    }
   ],
   "source": [
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed5e894-4597-4911-a623-591560f72b82",
   "metadata": {},
   "source": [
    "## Streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7311566b",
   "metadata": {},
   "source": [
    "Using `stream_complete` endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c896dbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = llm.stream_complete(\"Paul Graham is \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a79deaf3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Graham is a prominent computer scientist, entrepreneur, and venture capitalist, best known for co-founding the startup accelerator Y Combinator. He is also recognized for his essays on technology, startups, and programming, which have influenced many in the tech community. Graham has a background in programming languages and artificial intelligence and has authored several influential works, including \"Hackers and Painters.\" His insights on entrepreneurship and innovation have made him a respected figure in Silicon Valley."
     ]
    }
   ],
   "source": [
    "for r in resp:\n",
    "    print(r.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ae9c4a",
   "metadata": {},
   "source": [
    "Using `stream_chat` endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "173d8412",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"system\", content=\"You are a pirate with a colorful personality\"\n",
    "    ),\n",
    "    ChatMessage(role=\"user\", content=\"What is your name\"),\n",
    "]\n",
    "resp = llm.stream_chat(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cc4cb7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ahoy there! Ye can call me Captain Jollybeard, the most colorful pirate to sail the seven seas! What brings ye to me ship today?"
     ]
    }
   ],
   "source": [
    "for r in resp:\n",
    "    print(r.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87356c8d",
   "metadata": {},
   "source": [
    "## Configure Parameters\n",
    "\n",
    "The Respones API supports many options:\n",
    "- Setting the model name\n",
    "- Generation parameters like temperature, top_p, max_output_tokens\n",
    "- enabling built-in tool calling\n",
    "- setting the resoning effort for O-series models\n",
    "- tracking previous responses for automatic conversation history\n",
    "- and more!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9abef5c",
   "metadata": {},
   "source": [
    "### Basic Parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53e9ae5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    temperature=0.5,  # default is 0.1\n",
    "    max_output_tokens=100,  # default is None\n",
    "    top_p=0.95,  # default is 1.0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42974a34",
   "metadata": {},
   "source": [
    "### Built-in Tool Calling\n",
    "\n",
    "The responses API supports built-in tool calling, which you can read more about [here](https://platform.openai.com/docs/guides/tools?api-mode=responses).\n",
    "\n",
    "Configuring this means that the LLM will automatically call the tool and use it to augment the response.\n",
    "\n",
    "Tools are defined as a list of dictionaries, each containing settings for a tool.\n",
    "\n",
    "Below is an example of using the built-in web search tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff7a3aaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: As of 12:18 AM on Friday, March 28, 2025, in San Francisco, the current weather is partly sunny with a temperature of 61°F (16°C).\n",
      "\n",
      "## Weather for San Francisco, CA:\n",
      "Current Conditions: Partly sunny, 61°F (16°C)\n",
      "\n",
      "Daily Forecast:\n",
      "* Thursday, March 27: Low: 52°F (11°C), High: 61°F (16°C), Description: Periods of rain and drizzle beginning in the late morning; breezy this afternoon\n",
      "* Friday, March 28: Low: 47°F (8°C), High: 61°F (16°C), Description: A shower in the area in the morning; otherwise, clouds giving way to some sun\n",
      "* Saturday, March 29: Low: 50°F (10°C), High: 60°F (15°C), Description: Mostly sunny\n",
      "* Sunday, March 30: Low: 51°F (11°C), High: 59°F (15°C), Description: Cloudy; periods of rain in the morning followed by a shower in spots in the afternoon\n",
      "* Monday, March 31: Low: 49°F (10°C), High: 58°F (14°C), Description: Cloudy and cool; a couple of showers in the afternoon\n",
      "* Tuesday, April 01: Low: 53°F (12°C), High: 58°F (14°C), Description: Some sunshine giving way to clouds, breezy and cool; occasional rain in the afternoon\n",
      "* Wednesday, April 02: Low: 52°F (11°C), High: 56°F (13°C), Description: A couple of showers in the morning; otherwise, cloudy and remaining cool\n",
      "\n",
      "\n",
      "In March, San Francisco typically experiences daytime temperatures around 61°F (16°C) and nighttime temperatures around 47°F (8°C). The city usually receives about 3.5 inches (89 mm) of rainfall over approximately 11 days during the month. ([weather2visit.com](https://www.weather2visit.com/north-america/united-states/san-francisco-march.htm?utm_source=openai)) \n",
      "================\n",
      "{'built_in_tool_calls': [ResponseFunctionWebSearch(id='ws_67e5eaecce088191ab2edce452ef25420a24041ef7e917b2', status='completed', type='web_search_call')], 'annotations': [AnnotationURLCitation(end_index=1561, start_index=1439, title='San Francisco Weather in March 2025 | United States Averages | Weather-2-Visit', type='url_citation', url='https://www.weather2visit.com/north-america/united-states/san-francisco-march.htm?utm_source=openai')], 'usage': ResponseUsage(input_tokens=327, output_tokens=462, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=789, input_tokens_details={'cached_tokens': 0})}\n"
     ]
    }
   ],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    built_in_tools=[{\"type\": \"web_search_preview\"}],\n",
    ")\n",
    "\n",
    "resp = llm.chat(\n",
    "    [ChatMessage(role=\"user\", content=\"What is the weather in San Francisco?\")]\n",
    ")\n",
    "print(resp)\n",
    "print(\"========\" * 2)\n",
    "print(resp.additional_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78037c9e",
   "metadata": {},
   "source": [
    "## Reasoning Effort\n",
    "\n",
    "For O-series models, you can set the reasoning effort to control the amount of time the model will spend reasoning.\n",
    "\n",
    "See the [OpenAI API docs](https://platform.openai.com/docs/guides/reasoning?api-mode=responses) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6e31377",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: The question “What is the meaning of life?” has been asked by philosophers, theologians, scientists, and countless individuals throughout history—and there isn’t one definitive answer that satisfies everyone. Here are a few perspectives that help illustrate why the answer is so open-ended:\n",
      "\n",
      "1. Philosophical and Existential Views:  \n",
      " • Some existentialist thinkers, such as Jean-Paul Sartre and Albert Camus, argue that life doesn’t come with a preordained meaning. Instead, they suggest that it’s up to each individual to create their own purpose through choices, relationships, and personal projects.  \n",
      " • Other philosophies, like those in classical philosophy, have emphasized the pursuit of virtue, knowledge, or happiness as key to a meaningful life.\n",
      "\n",
      "2. Religious and Spiritual Perspectives:  \n",
      " • Many religious traditions offer meanings tied to their spiritual beliefs. In these views, life might be seen as a journey of growth, moral development, or fulfilling a divine plan—whether that means living according to God’s commandments, achieving enlightenment, or learning lessons through reincarnation.  \n",
      " • Spiritual traditions often encourage adherents to find meaning in love, compassion, and service to others.\n",
      "\n",
      "3. Scientific and Naturalistic Approaches:  \n",
      " • From a biological standpoint, one might say that the purpose of life is simply to survive and reproduce—the basic mechanisms that drive natural selection and evolution.  \n",
      " • Some argue that while biology defines life’s mechanics, human consciousness allows us to transcend mere survival and imbue our existence with personal and cultural meaning.\n",
      "\n",
      "4. Personal and Cultural Interpretations:  \n",
      " • For many, meaning is found in everyday connections: love, creativity, learning, and contributing to the community or society at large.  \n",
      " • Different cultures and individuals may prioritize various values (such as achievement, compassion, or exploration), which means the “meaning” can vary widely based on one’s upbringing, experiences, and personal reflections.\n",
      "\n",
      "5. A Humorous Take:  \n",
      " • Popular culture, notably in Douglas Adams’ The Hitchhiker’s Guide to the Galaxy, famously suggests that the answer to the ultimate question of life is “42”—a playful reminder that the real significance often lies in the journey of questioning itself rather than arriving at a single answer.\n",
      "\n",
      "In the end, the meaning of life might be considered less of an absolute truth and more of an invitation to explore, question, and define what matters most to you personally. Whether through relationships, creative pursuits, intellectual challenges, or spiritual practices, many believe we have the power to create our own meaning in a vast and complex universe.\n",
      "================\n",
      "{'built_in_tool_calls': [], 'reasoning': ResponseReasoningItem(id='rs_683e2dde0e308198a72c5e7f2e9bf52a0dd2faa5908183b8', summary=[], type='reasoning', encrypted_content=None, status=None), 'annotations': [], 'usage': ResponseUsage(input_tokens=13, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=841, output_tokens_details=OutputTokensDetails(reasoning_tokens=320), total_tokens=854)}\n"
     ]
    }
   ],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"o3-mini\",\n",
    "    reasoning_options={\"effort\": \"high\"},\n",
    ")\n",
    "\n",
    "resp = llm.chat(\n",
    "    [ChatMessage(role=\"user\", content=\"What is the meaning of life?\")]\n",
    ")\n",
    "print(resp)\n",
    "print(\"========\" * 2)\n",
    "print(resp.additional_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5170f908",
   "metadata": {},
   "source": [
    "## Image Support\n",
    "\n",
    "OpenAI has support for images in the input of chat messages for many models.\n",
    "\n",
    "Using the content blocks feature of chat messages, you can easily combone text and images in a single LLM prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9ac4a61",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cdn.pixabay.com/photo/2016/07/07/16/46/dice-1502706_640.jpg -O image.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4da96618",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The image shows three white dice with black dots, captured in mid-air above a checkered surface. The dice are in various orientations, displaying different numbers of dots. The background is dark, with a subtle light illuminating the dice, creating a dramatic effect. The checkered surface resembles a chess or checkerboard.\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.llms import ChatMessage, TextBlock, ImageBlock\n",
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(model=\"gpt-4o\")\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"user\",\n",
    "        blocks=[\n",
    "            ImageBlock(path=\"image.png\"),\n",
    "            TextBlock(text=\"Describe the image in a few sentences.\"),\n",
    "        ],\n",
    "    )\n",
    "]\n",
    "\n",
    "resp = llm.chat(messages)\n",
    "print(resp.message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90f07f7e-927f-47a2-9797-de5a86d61e1f",
   "metadata": {},
   "source": [
    "## Using Function/Tool Calling\n",
    "\n",
    "OpenAI models have native support for function calling. This conveniently integrates with LlamaIndex tool abstractions, letting you plug in any arbitrary Python function to the LLM.\n",
    "\n",
    "In the example below, we define a function to generate a Song object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "212bb2d2-2bed-4188-85ad-3cd497d4b864",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel\n",
    "from llama_index.core.tools import FunctionTool\n",
    "\n",
    "\n",
    "class Song(BaseModel):\n",
    "    \"\"\"A song with name and artist\"\"\"\n",
    "\n",
    "    name: str\n",
    "    artist: str\n",
    "\n",
    "\n",
    "def generate_song(name: str, artist: str) -> Song:\n",
    "    \"\"\"Generates a song with provided name and artist.\"\"\"\n",
    "    return Song(name=name, artist=artist)\n",
    "\n",
    "\n",
    "tool = FunctionTool.from_defaults(fn=generate_song)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daec99fe",
   "metadata": {},
   "source": [
    "The `strict` parameter tells OpenAI whether or not to use constrained sampling when generating tool calls/structured outputs. This means that the generated tool call schema will always contain the expected fields.\n",
    "\n",
    "Since this seems to increase latency, it defaults to false."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba29ea39-39cd-42f4-934e-78fba4935a1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name='Chasing Stars' artist='Luna Sky'\n"
     ]
    }
   ],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(model=\"gpt-4o-mini\", strict=True)\n",
    "response = llm.predict_and_call(\n",
    "    [tool],\n",
    "    \"Write a random song for me\",\n",
    "    # strict=True  # can also be set at the function level to override the class\n",
    ")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "629bdb64-e5bd-4539-9537-54352dd8dbb4",
   "metadata": {},
   "source": [
    "We can also do multiple function calling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c098cb38-bfed-4907-8109-6810756f2ae5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Hey Jude', 'artist': 'The Beatles'}}, Output: name='Hey Jude' artist='The Beatles'\n",
      "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Let It Be', 'artist': 'The Beatles'}}, Output: name='Let It Be' artist='The Beatles'\n",
      "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Come Together', 'artist': 'The Beatles'}}, Output: name='Come Together' artist='The Beatles'\n",
      "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Yesterday', 'artist': 'The Beatles'}}, Output: name='Yesterday' artist='The Beatles'\n",
      "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Twist and Shout', 'artist': 'The Beatles'}}, Output: name='Twist and Shout' artist='The Beatles'\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAIResponses(model=\"gpt-4o-mini\")\n",
    "response = llm.predict_and_call(\n",
    "    [tool],\n",
    "    \"Generate five songs from the Beatles\",\n",
    "    allow_parallel_tool_calls=True,\n",
    ")\n",
    "for s in response.sources:\n",
    "    print(f\"Name: {s.tool_name}, Input: {s.raw_input}, Output: {str(s)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7552118f",
   "metadata": {},
   "source": [
    "### Manual Tool Calling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dade92c4",
   "metadata": {},
   "source": [
    "If you want to control how a tool is called, you can also split the tool calling and tool selection into their own steps.\n",
    "\n",
    "First, lets select a tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a76226a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llms import ChatMessage\n",
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(model=\"gpt-4o-mini\")\n",
    "\n",
    "chat_history = [ChatMessage(role=\"user\", content=\"Write a random song for me\")]\n",
    "\n",
    "resp = llm.chat_with_tools([tool], chat_history=chat_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9eb38e2",
   "metadata": {},
   "source": [
    "Now, lets call the tool the LLM selected (if any).\n",
    "\n",
    "If there was a tool call, we should send the results to the LLM to generate the final response (or another tool call!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43163a59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calling generate_song with {'name': 'Chasing Stars', 'artist': 'Luna Sky'}\n"
     ]
    }
   ],
   "source": [
    "tools_by_name = {t.metadata.name: t for t in [tool]}\n",
    "tool_calls = llm.get_tool_calls_from_response(\n",
    "    resp, error_on_no_tool_call=False\n",
    ")\n",
    "\n",
    "while tool_calls:\n",
    "    # add the LLM's response to the chat history\n",
    "    chat_history.append(resp.message)\n",
    "\n",
    "    for tool_call in tool_calls:\n",
    "        tool_name = tool_call.tool_name\n",
    "        tool_kwargs = tool_call.tool_kwargs\n",
    "\n",
    "        print(f\"Calling {tool_name} with {tool_kwargs}\")\n",
    "        tool_output = tool(**tool_kwargs)\n",
    "        chat_history.append(\n",
    "            ChatMessage(\n",
    "                role=\"tool\",\n",
    "                content=str(tool_output),\n",
    "                # most LLMs like OpenAI need to know the tool call id\n",
    "                additional_kwargs={\"call_id\": tool_call.tool_id},\n",
    "            )\n",
    "        )\n",
    "\n",
    "        resp = llm.chat_with_tools([tool], chat_history=chat_history)\n",
    "        tool_calls = llm.get_tool_calls_from_response(\n",
    "            resp, error_on_no_tool_call=False\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78d611c5",
   "metadata": {},
   "source": [
    "Now, we should have a final response!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c5864e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here's a song for you titled **\"Chasing Stars\"** by **Luna Sky**!\n",
      "\n",
      "### Chasing Stars\n",
      "\n",
      "**Verse 1**  \n",
      "In the midnight glow, we wander free,  \n",
      "With dreams like fireflies, lighting up the sea.  \n",
      "Whispers of the night, calling out our names,  \n",
      "Together we’ll ignite, this wild, untamed flame.\n",
      "\n",
      "**Chorus**  \n",
      "We’re chasing stars, through the endless night,  \n",
      "With every heartbeat, we’ll take flight.  \n",
      "Hand in hand, we’ll break the dark,  \n",
      "In this cosmic dance, we’ll leave our mark.\n",
      "\n",
      "**Verse 2**  \n",
      "Underneath the moon, secrets softly shared,  \n",
      "Every glance a promise, every touch a dare.  \n",
      "The universe is ours, let the journey start,  \n",
      "With every step we take, we’re painting art.\n",
      "\n",
      "**Chorus**  \n",
      "We’re chasing stars, through the endless night,  \n",
      "With every heartbeat, we’ll take flight.  \n",
      "Hand in hand, we’ll break the dark,  \n",
      "In this cosmic dance, we’ll leave our mark.\n",
      "\n",
      "**Bridge**  \n",
      "And when the dawn arrives, we’ll still be here,  \n",
      "With the echoes of our laughter, crystal clear.  \n",
      "No matter where we go, no matter how far,  \n",
      "Forever in our hearts, we’ll chase those stars.\n",
      "\n",
      "**Chorus**  \n",
      "We’re chasing stars, through the endless night,  \n",
      "With every heartbeat, we’ll take flight.  \n",
      "Hand in hand, we’ll break the dark,  \n",
      "In this cosmic dance, we’ll leave our mark.\n",
      "\n",
      "**Outro**  \n",
      "So let’s chase the stars, let’s light the way,  \n",
      "In this beautiful journey, we’ll never stray.  \n",
      "With dreams as our compass, love as our guide,  \n",
      "Together we’ll soar, side by side.\n",
      "\n",
      "Feel free to let me know if you'd like any changes or another song!\n"
     ]
    }
   ],
   "source": [
    "print(resp.message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ede8d94-524b-4a51-8150-552df952f1bf",
   "metadata": {},
   "source": [
    "## Structured Prediction\n",
    "\n",
    "An important use case for function calling is extracting structured objects. LlamaIndex provides an intuitive interface for converting any LLM into a structured LLM - simply define the target Pydantic class (can be nested), and given a prompt, we extract out the desired object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04312089-bf9a-48d0-918f-ca1b3808439b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "from llama_index.core.prompts import PromptTemplate\n",
    "from pydantic import BaseModel\n",
    "from typing import List\n",
    "\n",
    "\n",
    "class MenuItem(BaseModel):\n",
    "    \"\"\"A menu item in a restaurant.\"\"\"\n",
    "\n",
    "    course_name: str\n",
    "    is_vegetarian: bool\n",
    "\n",
    "\n",
    "class Restaurant(BaseModel):\n",
    "    \"\"\"A restaurant with name, city, and cuisine.\"\"\"\n",
    "\n",
    "    name: str\n",
    "    city: str\n",
    "    cuisine: str\n",
    "    menu_items: List[MenuItem]\n",
    "\n",
    "\n",
    "llm = OpenAIResponses(model=\"gpt-4o-mini\")\n",
    "prompt_tmpl = PromptTemplate(\n",
    "    \"Generate a restaurant in a given city {city_name}\"\n",
    ")\n",
    "# Option 1: Use `as_structured_llm`\n",
    "restaurant_obj = (\n",
    "    llm.as_structured_llm(Restaurant)\n",
    "    .complete(prompt_tmpl.format(city_name=\"Dallas\"))\n",
    "    .raw\n",
    ")\n",
    "# Option 2: Use `structured_predict`\n",
    "# restaurant_obj = llm.structured_predict(Restaurant, prompt_tmpl, city_name=\"Miami\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b6fffcd-ff1e-4755-a851-1c6757a8075e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Restaurant(name='Tex-Mex Delight', city='Dallas', cuisine='Tex-Mex', menu_items=[MenuItem(course_name='Tacos', is_vegetarian=False), MenuItem(course_name='Vegetarian Enchiladas', is_vegetarian=True), MenuItem(course_name='Fajitas', is_vegetarian=False), MenuItem(course_name='Chips and Salsa', is_vegetarian=True), MenuItem(course_name='Queso Dip', is_vegetarian=True)])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "restaurant_obj"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df5fa1ab-f598-46da-80f3-f6af5dd2fe83",
   "metadata": {},
   "source": [
    "## Async"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e088b90-12b6-4211-a9ca-696e9897e9ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(model=\"gpt-4o\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f649683-896a-439e-b14b-e5df5d803b82",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = await llm.acomplete(\"Paul Graham is \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d05d8b7c-07b3-4ce8-9a8c-fcd1e830316c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Graham is a British-American entrepreneur, venture capitalist, and essayist. He is best known for co-founding Viaweb, one of the first web-based applications, which was later sold to Yahoo and became Yahoo Store. Graham is also a co-founder of Y Combinator, a highly influential startup accelerator that has funded and supported numerous successful startups, including Dropbox, Airbnb, and Reddit. In addition to his work in technology and startups, Graham is known for his insightful essays on topics such as programming, entrepreneurship, and the philosophy of work.\n"
     ]
    }
   ],
   "source": [
    "print(resp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abc72d09-3bcd-4d48-9bb7-0920c56310c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = await llm.astream_complete(\"Paul Graham is \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b28d7a06-1518-4ec0-b3cc-6364395b3561",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Graham is a British-American entrepreneur, venture capitalist, and essayist. He is best known for co-founding Viaweb, one of the first web-based applications, which was later sold to Yahoo and became Yahoo Store. Graham is also a co-founder of Y Combinator, a highly influential startup accelerator that has funded and supported numerous successful startups, including Dropbox, Airbnb, and Reddit. In addition to his work in technology and startups, Graham is known for his insightful essays on topics related to entrepreneurship, technology, and society."
     ]
    }
   ],
   "source": [
    "async for delta in resp:\n",
    "    print(delta.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3782e752-3c65-4b11-b2c6-8efe55fc33b1",
   "metadata": {},
   "source": [
    "Async function calling is also supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dcfaa77-edfe-42f9-8138-884c99dd3e62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name='Chasing Stars' artist='Luna Sky'\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAIResponses(model=\"gpt-4o-mini\")\n",
    "response = await llm.apredict_and_call([tool], \"Generate a random song\")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3d07bd",
   "metadata": {},
   "source": [
    "## Additional kwargs\n",
    "\n",
    "If there are additional kwargs not present in the constructor, you can set them at a per-instance level with `additional_kwargs`.\n",
    "\n",
    "These will be passed into every call to the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2da534d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4o-mini\", additional_kwargs={\"user\": \"your_user_id\"}\n",
    ")\n",
    "resp = llm.complete(\"Paul Graham is \")\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wV_g3KkE-Ty0",
   "metadata": {},
   "source": [
    "## Image generation\n",
    "\n",
    "You can use [image generation](https://platform.openai.com/docs/guides/image-generation?image-generation-model=gpt-image-1#generate-images) by passing, as a built-in-tool, `{'type': 'image_generation'}` or, if you want to enable streaming, `{'type': 'image_generation', 'partial_images': 2}`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ZDJhjBig_1Zq",
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "from llama_index.llms.openai import OpenAIResponses\n",
    "from llama_index.core.llms import ChatMessage, ImageBlock, TextBlock\n",
    "\n",
    "# run without streaming\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4.1-mini\", built_in_tools=[{\"type\": \"image_generation\"}]\n",
    ")\n",
    "messages = [\n",
    "    ChatMessage.from_str(\n",
    "        content=\"A llama dancing with a cat in a meadow\", role=\"user\"\n",
    "    )\n",
    "]\n",
    "response = llm.chat(\n",
    "    messages\n",
    ")  # response = await llm.achat(messages) for an async implementation\n",
    "for block in response.message.blocks:\n",
    "    if isinstance(block, ImageBlock):\n",
    "        with open(\"llama_and_cat_dancing.png\", \"wb\") as f:\n",
    "            f.write(bas64.b64decode(block.image))\n",
    "    elif isinstance(block, TextBlock):\n",
    "        print(block.text)\n",
    "\n",
    "# run with streaming\n",
    "llm_stream = OpenAIResponses(\n",
    "    model=\"gpt-4.1-mini\",\n",
    "    built_in_tools=[{\"type\": \"image_generation\", \"partial_images\": 2}],\n",
    ")\n",
    "response = llm_stream.stream_chat(\n",
    "    messages\n",
    ")  # response = await llm_stream.asteam_chat(messages) for an async implementation\n",
    "for event in response:\n",
    "    for block in event.message.blocks:\n",
    "        if isinstance(block, ImageBlock):\n",
    "            # block.detail contains the ID of the image\n",
    "            with open(f\"llama_and_cat_dancing_{block.detail}.png\", \"wb\") as f:\n",
    "                f.write(bas64.b64decode(block.image))\n",
    "        elif isinstance(block, TextBlock):\n",
    "            print(block.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Hb_aHUfWCMcN",
   "metadata": {},
   "source": [
    "## MCP Remote calls\n",
    "\n",
    "You can call any [remote MCP](https://platform.openai.com/docs/guides/tools-remote-mcp) through the OpenAI Responses API just by passing the MCP specifics as a built-in tool to the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "GOEjqNaoCeOt",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4.1\",\n",
    "    built_in_tools=[\n",
    "        {\n",
    "            \"type\": \"mcp\",\n",
    "            \"server_label\": \"deepwiki\",\n",
    "            \"server_url\": \"https://mcp.deepwiki.com/mcp\",\n",
    "            \"require_approval\": \"never\",\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "messages = [\n",
    "    ChatMessage.from_str(\n",
    "        content=\"What transport protocols are supported in the 2025-03-26 version of the MCP spec?\",\n",
    "        role=\"user\",\n",
    "    )\n",
    "]\n",
    "response = llm.chat(messages)\n",
    "# see the textual output\n",
    "print(response.message.content)\n",
    "# see the MCP tool call\n",
    "print(response.raw.output[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Y4RZvuXwEQFy",
   "metadata": {},
   "source": [
    "## Code interpreter\n",
    "\n",
    "You can use the [Code Interpreter](https://platform.openai.com/docs/guides/tools-code-interpreter) just by setting, as a built-in tool, `\"type\": \"code_interpreter\", \"container\": { \"type\": \"auto\" }`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "kOMpxzckEPeg",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAIResponses\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "llm = OpenAIResponses(\n",
    "    model=\"gpt-4.1\",\n",
    "    built_in_tools=[\n",
    "        {\n",
    "            \"type\": \"code_interpreter\",\n",
    "            \"container\": {\"type\": \"auto\"},\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "messages = messages = [\n",
    "    ChatMessage.from_str(\n",
    "        content=\"I need to solve the equation 3x + 11 = 14. Can you help me?\",\n",
    "        role=\"user\",\n",
    "    )\n",
    "]\n",
    "response = llm.chat(messages)\n",
    "# see the textual output\n",
    "print(response.message.content)\n",
    "# see the MCP tool call\n",
    "print(response.raw.output[0])"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "llama-index-caVs7DDe-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
